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Efficient management of the Large Area Crop Inventory 
Experiment (LACIE) dictates that effective controls of 
project activities be established. To provide a basis for 
effective control, documentation will be prepared, baselines 
will be established, and changes to the baseline will be 
subsequently controlled by the proper management levels. 

The specific control documents which will be used are 
defined in the LACIE Project Plan, document LAP01. All 
elements of the LACIE Project must adhere to these baselined 
control documents, and where it is considered that the 
requirements should be changed, the proper change request, 
accompanied by a full justification, must be submitted to the 
proper management level in accordance with established pro- 
cedures. These documents will be maintained current by change 
notices and revisions, as required. Each change notice and/or 
revision will reference the applicable Change Control Board 
Directive (CCBD) which approved the change. 

This document, LACIE-00200, Volume IX, defines the LACIE 
Test and Evaluation (T&E) requirements and has been prepared 
in accordance with the "Instructions for Preparation of LACIE 
Requirements Documents," LACIE-00100, Revision C, dated 
November 20, 197 1 *. Full-up System as used in this document 
is defined as the system required to accomplish LACIE II. In 
general the approach used in each section is to first specify 
the requirements of the Full-up System and then to specify 
the requirements of any interim systems by reference to 
specific paragraphs in the Full-up System requirements sections 
of the document. The LACIE Project phases are defined in the 
LACIE Project Plan, LAP01. 


The organization responsible for the implementation of 
each requirement defined in this document is specified on an 
individual requirement basis. Whe^e the implementation re- 
sponsibility applies to the complete section, the implementa- 
tion responsibility is specified after the section title. A 
section for the purpose of designating implementation respon- 
sibility is defined as being any numbered paragraph and all 
subparagraphs. Where different implementation responsibilities 
apply to different portions of a section, the implementation 
responsibility is specified on an individual paragraph basis 
as applicable. All implementing organizations designated shall 
accomplish their implementation activities in accordance with 
the requirements specified herein. 


tne requirements specifi 
R. B. MacDonald 


B. MacDona. 

Manager, Large Area Crop Inventory Experiment 
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1.0 FUNCTIONAL RESPONSIBILITIES 


1 . 1 GENERAL 


The responsibility of the Test and Evaluation Section 
(TES) is to evaluate techniques proposed for the LACIE 
according to coordinated experimental designs and to 
identify potential problem areas. Prior to the operation 
of a LACIE phase, TES will test proposed techniques and 
evaluate alternative designs. During and after the operation 
of a LACIE phase, TES will perform certain post-analysis 
tests (not real time) to Identify problem areas. 


1.2 SPECIFIC 


The specific tasks which comprise the TES responsibilities 
are discussed in section 4.0. 
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2.0 APPLICABLE DOCUMENTS 


The following document is applicable to the extent 
specified herein: Project Plan for the Large Area Crop 

Inventory Applications System Verification Test Experiment 
(DAPTS/LACIE) , Document GSPC, April 1974. DAPTS is the 
Data Acquisition, Preprocessing and Transmission Subsystem. 
GSFC is the Goddard Space Plight Center. 
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3.0 FUNCTIONAL FLOW DIAGRAMS 


3. 


3. 


PRECEDING ? 


3 . 1 DEVELOPMENT 
Not applicable. 

3 . 2 OPERATION 

2.1 Full-up System 
See figure 1. 

2.2 Interim System 
Not applicable. 
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SUBSYSTEM REQUIREMENTS 
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POST- 

ANALYSIS 




H.O REQUIREMENTS ( CONSOLIDATED ) 
(Imp 1 . RespTl LACIE/R t f£fe5 

4 . 1 PULL-UP SYSTEM 


To satisfy the objectives stated in section 1.1, the 
following tasks are to be completed in accordance with the 
stated task requirements. 


4.1.1 Classifier and Analyst Interpreter (AI) Performance 
(Required by CAMS; Category 1) 


This task shall estimate the performance of the Phase 1 
Classification and Mensuration Subsystem (CAMS) design 
relative to itself and provide a recommendation for the use 
of specific alternative design configurations. Specific 
evaluation tasks to be included are stated below. For tasks 

4.1.1. 1 through 4. 1.1. 5, multipass data for the first Earth 
Resources Technology Satellite (ERTS-1) covering the four 
biological phases specified by CAMS is to be used. 


4. 1.1.1 Formulate experiment design . The first task 
shall be to formulate the experimental design. This design 
shall define the procedure to follow In carrying out tasks 

4. 1.1.2 through 4. 1.1. 6 and shall specify the analvsis-of- 
variance (ANOVA) design to be used In analyzing the results. 


4. 1.1. 2 Estimate AI field labeling and boundary 
coordinate selection errors . Using the AI procedures 
proposed for the Phase 1 CAMS design, estimate the average 
AI error in labeling training fields as wheat or nonwheat, 
and estimate the average AI error in specifying training 
field coordinates. 


4. 1.1. 3 Estimate the probability of mlsclasslf ication 
(PMC) and proportional mean square error (PMSE) in the 
acreage estim ate. Estimate the PMC and PMSE in the classified 
wheat acreage per site for the nine design alternatives in the 
Phase 1 CAMS design. These estimates are to be made using AI- 
selected training fields. The design which gives the best 
performance for each site shall be stated. 
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4. l.l. 4 Estimate the total effect of AI training field 
selection on CAMS performance . Repeat task 4.1.3 substituting 
randomly selected training fields for AI selected training 
fields. 


4. 1.1. 5 Estimate the effect of AI training field 
selection on CAMS performance with labeling and field 
boundary errors removed . Correct any mislabeling errors 
and field boundary coordinate errors in the Al-selected 
training sample, and reestimate the PMC and PMSE in the 
wheat acreage using the classifier configuration that gave 
the best performance in task 4.1.4. 


4. 1.1. 6 Estimate classifier performance on single-pass 
ERTS data . Using the clustering routine that gives the best 
performance in task 4. 1.1. 4, estimate the PMC and PMSE in the 
wheat acreage estimate for each single pass of data. This 
shall be done using randomly selected training samples. 

4.1.2 Signature Extension 
(Required by CAMS; Category 1) 


This task shall estimate the performance of the Phase 1 
CAMS design in local signature extension applications. 
Specific tasks shall be as follows: 


4. 1.2.1 Formulate experimental design . This first 
task shall be to formulate the experimental design. This 
design shall define the procedures to follow in carrying 
out tasks 4. 1.2.1 and 4. 1.2. 3, shall specify the factors 
which are to be tested, and shall specify the ANOVA design 
to be used in analyzing the results. In formulating the 
design, consideration shall be given to the following factors 
to determine whether or not they significantly affect 
signature extension: 

a. The spatial and temporal distance of the extension 

b. The soil type (more specifically, the soil color) 

c. The soil moisture 
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d. In single-pass extensions, the biological growth 
phase at the time of data acquisition 

e. The exclusion or inclusion of a sun angle correction 

f. The exclusion or Inclusion of a mean level adjustment 


4. 1.2.2 Extend single-pass signatures . This task shall 
estimate the c tiange i n t h e PMC in local signature extensions 
by estimating the PMC in an intensive site and In neighboring 
Statistical Reporting Service (SRS) sites. The training 
statistics are to be computed from samples drawn by a random 
selecting procedure over the intensive site. Single-pass 
ERTS data from each of the four wheat-growth biological 
phases (as specified by CAMS) are to be used. The CAMS 
classifier configuration, consisting of the ISOCLS clustering 
routine and the two-class classifier, shall be used. 


4. 1.2. 3 Extend multipass signatures . This task is 
similar to task 4. 1.2. 2 except that the single-pass ERTS 
data used in the preceding task Is to be composed, and this 
multipass data is to be used In place of the above single- 
pass data. In addition to the CAMS configuration, the feature 
selection without replacement routine specified by CAMS is to 
be used. 


4.1.3 Thresholding Procedures 
(Required by CAMS; Category 1) 


This task shall develop a procedure for selecting 
threshold values in the Phase 1 CAMS two-class classifier 
and test this procedure on ERTS Intensive site data over 
Kansas. The thresholding procedure shall be developed with 
the design goal of minimizing the PMSE in the wheat acreage 
estimate. The procedure is to be evaluated according to an 
experimental design which Is to be formulated prior to the 
evaluation. This design Is to specify the evaluation steps 
and the ANOVA design to evaluate the test results. 
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4.1.4 Selection of Prior Probabilities 


This task shall develop a procedure for estimating a 
set of prior probabilities for the Phase 1 CAMS two-class 
classifier. The resulting estimates are to be tested on 
ERTS intensive site data over Kansas using the PMSE in wheat 
acreage estimate as the evaluation criterion. If the 
estimating procedure that is developed does not use historical 
data on wheat acreage, that estimate is to be compared with 
the estimate based on the historical data. This comparison 
is to be made by including the historical estimate in the 
above intensive-site-data evaluation. The prior probability 
estimates are to be evaluated according to an experimental 
design which is to be formulated prior to the evaluation. 

This design is to specify the evaluation steps and the ANOVA 
design to evaluate the test results. (Required by CAMS; 
Category 1.) 


4.1.5 Alternative Methods for Training a Classifier 
(Required by CAMS; Category 1) 

This task shall develop training sample selection 
procedures that are different from those to be used in the 
Phase 1 CAMS design and which offer a potential improvement. 
Specific tasks to be included are stated below. All testing 
is to be done using the Kansas intensive site data to be 
used in task 4.1. 


4. 1.5.1 Formulate experimental design . This design 
shall specify the steps to follow in testing all proposed 
training field selection procedures and shall specify the 
ANOVA design to be used in analyzing the results. 


4. 1.5*2 Evaluation of the use of a cluster map as an 
aid for selecting training samples . This task shall develop 
and evaluate procedures which use the spectral and spatial 
clustering maps to guide the selection of training samples. 
The procedures so developed are to be tested against the 
Phase 1 CAMS design training field selection procedure. 
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4 . 1 . 5 - 3 Evaluate the utility of classification decisions 
based on posterior probabilities . This task shall evaluate 
any potential Improvement to be gained by restricting the 
photointerpreter task to selecting just wheat-training 
samples and by using the posterior probability of having 
wheat as the decision rule rather than the likelihood ratio- 
rule that is used in the Phase 1 CAMS design. The procedure 
for selecting wheat-training samples shall be developed. This 
procedure shall have a design goal of minimizing the PMSE in 
the wheat estimate when the posterior probability decision 
rule is used. If the developed selection procedure differs 
from the selection procedure that selects wheat-training 
fields in the Phase 1 CAMS design, this new procedure should 
be tested against the CAMS procedure. 


4.1.6 Performance Analysis of LACIE Operations 
(Required by CAMS; Category 2) 


4. 1.6.1 Postoperation analysis . This task shall 
provide an experimental design for evaluating the performance 
of selected elements of the LACIE to support postanalysis (not 
in real time) of LACIE phases. This design is to specify the 
variables to be measured and the statistical approach to be 
used to analyze the results. 


4. 1.6. 2 Ad hoc analysis . This task shall provide an 
analysis of certain LACIE components which are identified 
during the operation of LACIE as potential trouble areas. 
This task shall provide a minimal level-of-effort analysis 
to take advantage of the LACIE operations to make measure- 
ments and to test in an off-line fashion, ad hoc concepts. 


4.1.7 Spatial Sampling Strategy Evaluation 
(Required by CAS; Category 2) 


This task shall evaluate the sampling plan that will be 
used in the Phase 1 design of the LACIE and the to-be- 
determined (TBD) alternatives to that plan. LACIE critical 
variables which are significantly affected by a given 
sampling plan are to be identified. Variables which are to 
be considered as potential candidates for this list shall 
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oe the probability of acquiring cloud-free data, photo- 
interpreter performance, and signature extension performance. 
Each plan is to be exercised on real or simulated data to 
validate the list of critical variables and to quantify 
the effect of the sampling plan on those variables. 


4.1.8 Comparison of Yield Models 
(Required by YES; Category 2) 


This task shall evaluate agrometeorology yield models. 
The first task shall be to formulate an experimental design. 
This design is to specify the factors over which the 
evaluation is to be performed and the ANOVA design to be 
used for the analysis of the results. 


4.1.9 Evaluation of Classification Techniques 
Which Incorporate Spatial Information 
(Required by CAMS; Category 2) 


Several candidate techniques have been identified for 
test and evaluation. 


4. 1.9.1 Environment Research Institute of Michigan 
(ERIM) multichannel decision rule . This task shall evaluate 
the nine-point classification rules that are defined in the 
National Aeronautics and Space Administration (NASA) report 
NASA-CR-ERIM 190100-32-T. The exact definitions of the 
rules to be evaluated are TBD. The procedure to be used 
for the evaluation, the ERTS data to be used, and the ANOVA 
design to analyze the results shall be part of the experi- 
mental design. This design is to be constructed by the LACIE 
Research, Test, and Evaluation Branch (RTEB). 


The contents of the final report are TBD by the LACIE 
RTEB. At the discretion of the Lyndon B. Johnson Space 
Center/Earth Observations Division ( JSC/EOD) , a complete 
documentation of the computer programs required to Implement 
the decision rules shall be provided. 
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it. 1.9.2 Rice University post-classification spatial 
filtering algorithm . This task shall evaluate the subject 
algorithm. First , an experimental design will be formulated 
which will specify the evaluation criteria, the significant 
factors, and the ANOVA design to be used for the analysis 
of results. 


4.1.10 Evaluation of the Kalman Filter 
in Signature Extension 
(Required by CAMS; Category 2) 

Th? s task shall evaluate the Kalman filter, as defined 
in NASA report NASA CR-ERIM 31650-155-T, for application in 
signature extension. The exact definition of the filter 
algorithm to be evaluated is TBD. The procedure to be used 
for evaluating the ERTS data to be used and the ANOVA design 
to analyze the results shall be part of the experimental 
design. This design is to be constructed by JSC/EOD. At 
the discretion of JSC/EOD, a complete documentation of the 
computer programs required to implement the filter shall be 
provided. 


4.1.11 Evaluation of System Error Models 
(Required by CAS; Category 2) 

This task shall evaluate all the system error models 
identified by the Crop Assessment Subsystem (CAS). The first 
task will be to formulate an experimental design. This design 
will specify the factors over which the evaluation is to be 
performed, the success criteria, and the ANOVA to be used for 
the analysis of results. (Required by CAS; Category 2.) 


4.1.12 Evaluation of Yield Models Relating 
Yield to Spectral and Spatial Appearance 
(Required by YES; Category 2) 

This task shall evaluate yield models relating yield 
to the spectral and spatial appearance of wheat. The first 
task will be to formulate an experimental design. This 
design will specify factors over which the evaluation is to 
be performed, the success criteria, and the ANOVA to be 
used for the analysis of results. 
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4.1.13 Evaluation of Cloud Detection 
and Data Dropout Algorithms 
(Required by CAMS/DAPTS; Category 2) 

This task shall evaluate cloud detection algorithms 
and also include data dropout detection algorithms. The 
first task will be to formulate an experimental design. 
This design will specify factors over which the evaluation 
is to be performed, the success criteria, and the ANOVA 
to be used for the analysis of results. 


4.1.14 Evaluation of Alternative Analyst 
Interpretation Techniques 
(Required by CAMS; Category 2) 

This task shall evaluate alternative procedures and/or 
techniques for performing analyst interpretation tasks. The 
first task will be to formulate an experimental design. 

This design will specify the factors over which the evalua- 
tion is to be performed, the success criteria, and the 
ANOVA to be used for the analysis of results. 


4.1.15 Evaluation of Interactive Mult ispectral Image 
Analysis System, Model 100 (IMAGE100) for LACIE Problem 
(Required by CAMS; Category 2) 

This task shall evaluate the utility of special-purpose 
computational systems for applicability to LACIE functions. 
This task shall be performed utilizing the IMAGE100 to be 
constructed by NASA/Science and Applications Directorate 
(S&AD)/Flight Support Office (FSO) and will include any 
subsequent augmentation. This task will be phased in 
accordance with the system design schedule. 


The first task shall be to formulate a general 
experimental design. Specific evaluation criteria will 
be specified for each TBD phase of the system implementation. 



4.1,16 Evaluation of Acreage Determination Techniques 
(Required by CAS; Category 2) 


This task shall evaluate procedures for calculating 
acreage. The first task will be to formulate an experimental 
design. This design will specify the factors over which the 
evaluation is to be performed, the success criteria, and the 
ANOVA to be used for the analysis of results. 


if. 1.17 Evaluation of Production Determination Techniques 
(Required by CAS; Category 2) 


This ta3k shall evaluate procedures for calculating 
production. The first task will be to formulate an 
experimental design. This design will specify the factors 
over which the evaluation is to be performed, the success 
criteria, and the ANOVA to be used for the analysis of 
results . 


4.1.18 Evaluation of Registration Techniques 
(Required by DAPTS/CAMS; Category 2) 

Several tasks related to registration have been 
identified which require a test and evaluation effort: 

a. Evaluation of alternate techniques for correlation/ 
registration 

b. Evaluation of techniques for full-frame registration 

c. Evaluation of interpolation schemes utilized for 
registration 

d. Evaluation, if required, of hardwired registration 
algorithms on the IMAGE100 


For all these tasks, an experimental design will be 
formulated which will specify the factors over which the 
evaluation is to be performed, the success criteria, and 
the ANOVA to be used for the analysis of results. 
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4.1.19 Evaluation of SR&T Tasks 
(Required by all Subsystems, Category 3) 


Pursuant to the SR&T program to support LACIE by the 
RTEB, TES shall have tasks to evaluate the SR&T task3 which 
have be-^n determined by a TBD criteria to add significant 
improvement to existing operational techniques, procedures, 
and/or processing capabilities. In these instances, a TBD 
design will be developed and implemented which will rigidly 
evaluate the output of an SR&T study as to its applicability 
to the LACIE operational system. As a particular consider- 
ation, the T&E test plan to evaluate techniques for 
selecting analog areas shall include a statistical test to 
compare the PMC/proportion estimate in a sample of pairs of 
analog sites when training statistics are derived from one 
and applied to the other. 


4.1.20 Investigation of Per Field Classifiers 
(Required by CAMS; Category 2) 


This task shall investigate the current state of the 
art related to per field classifiers and summarize the 
results of this investigation in a report. 


4.2 INTERIM SYSTEM 
(Required by CAMS; Category 2) 


During the operational utilization of the LACIE1 data 
system by CAMS, it is anticipated that procedures and/or 
techiques, which may or may not deviate from the delivery of 
the LACIE2 data system, could appear troublesome enough to 
warrant a quick-response evaluation effort. During this 
period, TES shall provide, within limits, some level-of- 
effort support for specialized interim test and evaluation 
studies . 
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5.0 INPUT REQUIREMENTS 


✓ 


Inputs required by T&E from the other LACIE functional 
elements are defined in the followinf subparagraphs. 


5.1 PULL-UP SYSTEM 

5.1.1 DAPTS 


5. 1.1.1 ERTS data requirements . 

a. The CAMS evaluation *<.1 shall be conducted on 
the intensive test sites (ITS) data using a 
maximum of four ERTS passes (one pass per 
biological window) or as many passes as needed 
to meet the cloud cover criterion of .10 percent 
or less. The following ITS shall be processed 
and in the following order: 

• Five Kansas sites 

• One Montana site 

• One Canadian site 

• One Indiana site 

Computer-compatible tapes (CCT's) for each of these 
sites for each of the biological windows (5.1.3) 
are required. (Required by T&E ; Category 2; 
Implementation Responsibility: DAPTS/NASA-GSPC . ) 

b. In order to select the optimum data for each ITS, 
the DAPTS is required to screen all ERTS passes 
over the ITS and generate a list of those data sets 
which fall with the biological windows as defined 
by 5. 1.3. a and which contain 10 percent or less 
cloud coverage over the site. Requests for data 

in screening shall be made on an as-needed basis; 
that is, site priority requirement changes, etc. 
Thus, the priority for data tapes/forms to be used 
for the screening function shall be: 
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• In-house CCT’s on the data analysis station 
(DAS) or equivalent 

• Actual ERTS imagery 

• GSPC microfilm 

• GSPC ERTS data listing 

(Required by T&E; Category 2; Implementing 
Responsibility: DAPTS.) 

c. Upon receipt of CCT's which arrive based on a 

retro-order, these tapes should be screened and 
edited, and film products generated by a TBD 
procedure. In the event that GSPC is unable to 
deliver edited registered tapes, an in-house 
capability to perform these functions will be 
required. The design for the interim capability 
will be different than that which will be utilized 
after GSPC becomes operational. However, the 
requirement for the general function of screening 
these incoming CCT’s will remain. Based on the 
results of this screening effort, the DAPTS is 
required to order and maintain a statUo log of 
all ERTS data acquired In support of the LACIE 
RTEB. (Required by T&E; Category 2; Implementa- 
tion Responsibility: DAPTS.) 

d. The DAPTS shall provide an acquisition history of 

attempts to acquire ERTS data. This is to ir lude 
an account of all data acquisition attempts tnat 
were aborted due to cloud cover, data dropout, or 
other acquisition malfunction. (Required by T&E; 
Category 2; Implementation Responsibility: DAPTS.) 

e. For each ITS, a reference base data set will be 
selected. As rated previously, data sets for each 
of the biological phases will be selected depending 
on cloud coverage, etc. Thus, it will be required 
that each CCT received be registered to its associ- 
ated reference data set. As a maximum, this will 
be nearly equal to 104 registration-operations 

(26 ITS * 4 phases). The reference set will be 
registered to conform with a 1:24,000 scale when 
output as standard computer listings. (Required by 
T&E; Category 2; Implementation Responsibility: 
DAPTS . ) 
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f. For signature extension test and evaluation (4.2), 
three approximately contiguous ERTS frames of 
Kansas, acquired about four times a year (avail- 
ability of four passes will depend upon cloud 
cover) during the four LACIE growing phases, will 
be used. These three frames are to each include 
at least one intensive site and TBD SRS sites. 

Each evaluation study will consider one intensive 
site along with its surrounding SRS sites. 

After acquiring the SRS data, these sites shall be 
plotted on an appropriate base (for example, 
1:250,000 topographical maps) along with the nearby 
intensive sites. An overlay of the available ERTS 
data utilized by the CAMS evaluation shall then be 
constructed depicting these ERTS frame boundaries. 
These materials can then be submitted for review 
by RTEB personnel for selection of the required 
data sets. (Required by T&E; Category 2; Implemen- 
tation Responsibility: DAPTS/NASA-S&AD-FSO). 

g. For other test and evaluations tasks (4.3 through 
4.12), additional ERTS-1 data products may be 
required. This data will include both full-frame 
and GSFC-reduced lOxll-mile areas. (Required by 
T&E; Category 3; Implementation Responsibility: 
(DAPTS/NASA-GSFC) . 


5. l.l. 2 Field data requirements . (Required by T&E; Cate- 
gory 2; Implementation Responsibility: DAPTS/NASA-S&AD-FSO). 

The requirement for field observations in the ITS Is the same 
as the 1973-74 crop year under the United States/Canada pro- 
ject as defined in LACIE-00200, Volume IB, paragraph 4,1.3 
and 4.1.4, with the following exceptions: 

a. An additional wall-to-wall land-use inventory is 
required over the winter wheat and mixed winter 
and spring wheat test sites. This inventory shall 
take place after the fall planting of winter wheat 
and prior to the dormant period In the 1974 crop 
year . 

b. The format utilized for the 18-day periodic obser- 
vations will be modified in the manner In which the 
gorwth stages are recorded. The field observations 
form will utilize the terminology specified by the 
Yield Estimate Subsystem (YES) TBD in 5-1*3- 
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c. Controlled mosaics and overlays for each of the 
TBD intensive sites are required. In addition 
controlled mosaics and overlays covering each TBD 
SRS site are also required. 

A controlled mosaic at 1:24,000 scale covering the 
intensive test site is required. For this mosaic 
an overlay is to be made on which each field 
boundary is drawn and indexed. A listing corres- 
ponding to the field indices is to be provided in 
which the crop planted in each field is identified 
in TBD detail. All other areas not included in 
the field boundaries are also to be Indexed and 
described in TBD detail. 

d. Ground-truth overlays keyed to the aforementioned 

mosaics are required. Each field Identified in 
the fall enumeration (section 1.6.12) must be 
annotated on the overlay. A color-coding or gray 
shading should be used to show visual separation 
between the following five categories: winter 

wheat, spring wheat, other small grain, other 
agricultural, nonagricultural . 

e. Topographic maps are required for each ITS in 
identification, determination of relief, and 
identification of manmade features. The scales 
for these maps (in order of preference) shall be 
1:24,000, 1:62,000, or 1:250,000. 

5. 1.1. 3 Historical agricultural requirements . 


a. Historic SRS agricultural estimates for 1973 are 
required for all United States counties which 
contain at least one intensive study site. 
Cropping practices reports will be required for 
each segment to be used to determine: 

• What crops have historically been grown in 
the ITS? 

• What percent of the segment Is planted In each 
crop? 

• What shapes are the fields? 

• Do the fields have fallow strips? 
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• What percent of irrigation is present? 

• What fertilizers are used and how much is 
being used? 

(Required by T&E; Category 2; Implementation 
Responsibility : DAPTS/USDA) . 

b. Crop calendars for all intensive test sites are 
required. Specific calendars for each of the 
small-grain crops and all other major crops 
associated with each ITS will be required. 

(Required by T&E; Category 2; Implementation 
Responsibility : DAPTS/YES) . 

c. The percentage of wheat in the vicinity of the ITS 
under investigation for at least 1 year pervious 
to the data acquisition year is required. 

(Required by T&E; Category 2; Implementation 
Responsibility: DAPTS/USDA). 

d. For a TBD area in Kansas, the wheat acreage planted 
and harvested for all countries within TBD strata 
over a 10-year period is required. (Required by 
T&E; Category 2; Implementation Responsibility: 
DAPTS/USDA). 

e. The probability distributions for the TBD percentage 

of cloud coverage over TBD areas in Kansas as a 
function of time is required for the period of 
1973-197** • (Required by T&E; Category 3j Implemen- 
tation Responsibility: DAPTS) . 


5. 1.1.4 Real-time meteorological data requirements . 
Not applicable. 


5. 1.1.5 Historical meteorological data requirements . 
Weekly weather and crop bulletins shall be required for 
each ITS to determine how the 1973-7** year varied from the 
mean with respect to factors such as weather, fertilizer 
use, and percent of area in each crop. In addition, 
meteorological data from crop calendar updates shall be 
required. (Required by T&E; Category 3; Implementation 
Responsibility : DAPTS/YES/NOAA ) . 
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5.1.2 CAMS 


v 


> 


a. In order to optimally relate results from test 

and evaluation tasks to the LACIE operational 
systems, where applicable, T&E shall adhere to 
the same functional techniques and/or procedures 
utilized by these systems. Thus It Is required 
that the operational procedures and any sub- 
sequent revisions be provided for both the 
analyst Interpretation and automatic data pro- 
cessing operations. (Required by T&E; Category 1; 
Implementation Responsibility: CAMS). 

b. In addition to the software that already exists 
as part of the Algorithm Simulation Test and 
Evaluation Program (ASTEP), the following software 
is required: 

• Algorithm for Inserting nonrectangular field 
boundaries. 

• CAMS classification algorithm. 

• CAMS feature-selection algorithm for selection 
without replacement. 

• CAMS feature-selection algorithm for linear 
combinations using B-distance. 

• CAMS clustering algorithm for chaining of fields 
using B-distance. 

• Computer programming to include as a part of 
ASTEP a TBD mean-level correction algorithm. 

This algorithm should be derived using the 
ERIM-documented routine as a baseline. In 
addition, computer programming is required to 
include as a part of ASTEP a TBD routine for 
adjusting class statistics to account for 
variations in the sun angle. 

(Required by T&E; Category 2; Implementation 
Responsibility: C AMS/NAS A/DS AD) . 


\ 
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5.1.3 YES 


It Is required that the biological phases and their 
associated windows (start and end calendar dates) 
be defined for each ITS. For the winter wheat 
sites and mixed winter/spring wheat sites, bio- 
logical Phase 1 should be defined into two parts: 

• la - Plant date of winter wheat to beginning 
of dormancy 

• lb — Two weeks before end of dormancy to 
beginning of tillering. 

(Required by T&E; Category 2; Implementation 
Responsibility : YES/USDA) . 

The specific small-grain growth stages definition 
and their normal duration for each ITS is required; 
that is, a breakdown of small-grain phenology for 
monitoring purposes from emergence to harvest. 
(Required by T&E; Category 1; Implementation 
Responsibility : YES/USDA) . 

Updates to crop calendars for all the ITS are 
required, specific calendars for each of the small- 
grain crops and all other major crops associated 
with each ITS will be required as well as real- 
time updates. (Required by T&E; Category 2; 
Implementation Responsibility: YES). 

The description of candidate yield models along 
with associated algorithms, program documentation, 
and other TBD data will be required. (Required by 
T&E; Category 3; Implementation Responsibility: 
YES/NOAA) . 


5.1.4 
(Impl. Resp.: 


CAS 

CAS/RTEB) 


The descriptions of candidate production models 
and associated algorithms, program documentation, 
and other TBD data are required. 
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b. 


The descriptions of candidate error models and 
associated algorithms, program documentation, 
and other TBD data are required. (Required by 
T&E; Category 3). 


5.1.5 Information Storage, Retrieval, and Reformatting 

Subsystem (ISRRS) 


Not applicable. 


5.1.6 Systems Performance Evaluation (SPE) 


Not applicable. 


5.1.7 Information Evaluation (IE) 


Not applicable. 


5.1.8 Research 
(Impl. Resp.: LACIE/RTEB) 


a. The RTEB shall provide a yield model relating 
the spectral and spatial appearance of wheat to 
yield. (Required by T&E/YES; Category 2). 

b. It is required that the RTEB research and develop 
algorithms for automatic cloud and shadow detection 
and data dropout detection. (Required by T&E/CAMS; 
Category 2). 

c. It is required that the RTEB research and develop 
improved techniques for performing the analyst 
interpretation in LACIE. Candidate techniques 
should include utilizing linear combinations of 
multipass data and channel ratioing in both film 
and cathode ray tube (CRT) display mediums. 
(Required by T&E /CAMS; Category 2). 



5.1.9 Test and Evaluation 


Not applicable. 


5.2 INTERIM SYSTEMS 


Prior to the release of the LAC IE 2 operational system, 
the CAMS will be performing systems testing, debugging, and 
evaluation. It is required by T&E that complete documen- 
tation, where applicable, of all significant TBD factors be 
provided. (Required by T&E; Category 2; Implementation 
Responsibility: CAMS). 
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6.0 OUTPUT REQUIREMENTS 


Output requirements from T&E which will be provided 
to the other LACIE functional elements are defined in the 
following subparagraphs. 


6 . 1 PULL-UP SYSTEM 
( Impl . Resp . : LACIE/RTEB ) 


6.1.1 Classifier and AI Performance 

(Required by CAMS/CAS ; Category 2 ) 

This is the output requirement for T&E study 4.1.1. 

The output medium is a report written in accordance with 
the format specified in the experimental design. This 
report shall include: 

a. An estimate of the probability of mislabeling 
by the photointerpreter 

b. An estimate of the PMC and PMSE in the classified 
wheat acreage per site for the nine design 
alternatives for the classifier in CAMS, using 
photointerpreter-selected training fields. The 
report will also specify the design which gave 
the best performance. 

c. An estimate of the total effect of the photo- 
interpreter training field selection on CAMS 
performance. 

d. An estimate of the partial effect (labeling errors 
removed) on the classifier performance due to 
photointerpreter selection versus random selection 
of training samples. 

e. An estimate of the PMC and PMSE in the wheat 
acreage estimate for each single-pass data. Use 
randomly selected training samples. 

f. A description of the experimental design. 


PRECEDING PAGE BLANK NOT 


M T ZU 
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6.1.2 Signature Extension 
(Required by CAMS; Category 2) 

This is the output requirement for T&E study 4.1.2. 
The output medium is a report written in accordance with 
the format given in the experimental design. 


The report shall discuss analysis res ilts aimed at 
identifying factors which account for signature extension 
results. In particular the report shall discuss whether 
or not any of the following factors significantly influence 
signature extension: 

a. Distance of the extension 

b. Soil type 

c. Soil moisture 

d. Time of data acquisition in the extension site 

e. Biological growth phase at the time of data 
acquisition 

f. Exclusion or inclusion of a sun angle correction 

g. Exclusion or inclusion of a mean-level adjustment 


The report shall also include the estimated PMC in 
the training segment and in the extension segment. The 
report shall describe the experimental design. 


6.1.3 Thresholding Procedures 
(Required by CAMS; Category 2) 

This is the output requirement for T&E study 4.1.3. 

The output medium will be a report written in accordance 
with the format in the experimental design. This report 
shall include: 

a. A tabulation of the PMC and PMSE in the wheat acreage 
estimate versus the vector of class threshold values. 
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b. A statistical description of the class distributions. 
This is to Include the means of each class, the 
covariance matrix of each class, the Bhattacharaya 
distance between each pair of classes, and a scatter 
plot of a randomly selected subset of the points 
that were classified. The scatter plot is to 
Include the sample points selected from training. 

c. A description of the training sample selection 
procedure and results. 

d. A recommended procedure for selecting threshold 
values. 

e. A description of the classification algorithms used 
in the evaluation. 

f. A description of the experimental design. 


6.1.4 Selection of Prior Probabilities 
(Required by CAMS; Category 2) 

This is the output requirement for T&E study 4.1.4. 

The output medium shall be a report written in accordance 
with the format of the experimental design on the effect of 
using different priors on classification performance. 

Report topics are to include: 

a. A tabulation of the PMC and PMSE in the wheat 
acreage estimate versus the vector of class prior 
probabilities . 

b. An estimate of the PMC and PMSE in the classified 
wheat acreage per site for the nine design 
alternatives for the classifier in CAMS, using 
photointerpreter-selected training fields. The 
report will also specify the design which gave the 
best performance. 

c. An estimate of the total effect of the photo- 
interpreter training field selection on CAMS 
performance . 
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d. An estimate of the partial effect (labeling errors 
removed) on the classifier performance due to 
photointerpreter selection versus random selection 
of training samples. 

e. A recommended procedure for selecting prior 
probabilities for the classifier. 


6.1.5 Alternative Methods for Training a Classifier 
(Required by CAMSj Category 2) 


This is the output requirement for T&E study ^.1.5. 
The output medium is a report written in accordance with 
the format of the experimental design. This report shall 
Include : 


a. For the sample selection method based on the use of 
a cluster map, a tabulation of the PMC and the PMSE 
in the wheat acreage estimate versus the number of 
training samples selected in each of the classes of 
wheat and confusion crops. 

b. A comparison of the PMC and the PMSE in the wheat 
acreage estimate between the two-class likelihood 
ratio classification procedure, as defined for 
Phase 1 of CAMS, and the use of the posterior 
probability for a two-class classification. 

c. A description of the experiment design. 


6.1.6 Performance Analysis of LACIE Operations 
(Required by CAMS/CAS; Category 2) 


This is the output requirement for T&E study ^.1.6. 
The output medium for task a is a report written in 
accordance with the format defined in the experimental 
design. For task b, the output medium and format is TBD. 
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6.1.7 Spatial Sampling Strategy Evaluation 
(Required by CAMS/CAS; Category 2) 

This la the output requirement for T&E study 4.1.7. 

The output medium shall be a report. This report shall 
Inc lude : 

a. The estimated mean proportion of wheat over a TBD 
test area for each sampling plan that Is evaluated. 

b. The variance and bias of that estimate. 

c. An identification of LACXE variables which are 
significantly affected by the choice of the 
sampling plan under consideration. Variables 
which shall be considered include the probability 
of acquiring cloud-free data, photointerpreter 
performance, signature extension performance, and 
sensitivity to acreage and production calculations. 

d. A quantitative estimate of the extent to which these 
variables are affected by the choice of the sampling 
plan. 


6.1.8 Comparison of Yield Models 
(Required by YES; Category 3) 

This is the output requirement for T&E study 4.1.8. 
The output medium shall be a report that shall define the 
major factors which account for the variations in yield 
estimates as obtained from agrometeorology models. 
(Implementation Responsibility: T&E/LACIE RTEB/NOAA). 


6.1.9 Evaluation of Classification Techniques Which 
Incorporate Spatial Information 
(Required by CAMS; Category 2) 


This is the output requirement for T&E study 4.1.9. 
The output medium shall be reports which describe the 
results of the evaluations in accordance with the format 
of the experimental design. If the results of the ERIM 
nine-point classification study conducted by ERIM warrant 
it, a follow-on test and evaluation task will be conducted 



by TES by a TBD design. A report shall be written to 
describe the major considerations and environment under 
which a decrease In the PMC and PMSE can be achieved by 
this technique. Recommendations for Implementation 
considering operational constraints will be included If 
the magnitude of Improvement Is determined to be significant. 


If either the results of the ERIM study or the follow- 
on TES test and evaluation determine that no significant 
Improvement In the results under the operational LACIE 
environment Is feasible with this technique, the report 
will merely document the study results and specify that 
Implementation Is not recommended. 


6.1.10 Evaluation of the Kalman Filter in 
Signature Extension 
(Required by CAMS; Category 2) 


This is the output requirement for TIE study 4.1.10. 

The output medium shall be a report on the results of the 
ERIM evaluation of the utility of the Kalman filter in 
signature extension. If this technique is ineffective, 
a report which describes the basic study characteristics 
and results will be produced. If it is determined that the 
technique displayed sufficient increase in performance 
under the study conditions, then a follow-on test and 
evaluation study will be conducted by TES. Based on the 
results of this study, a report will be generated which 
lists the results under a TBD design and either describes 
the major faults of the technique or recommends the 
Implementation of it into the LACIE operational system. 

In the latter case, additional information will be described, 
including algorithms, program documentation, constraints, 
and other TBD Information. 


6.1.11 Evaluation of System Error Models 
(Required by CAS; Category 3) 

This is the output requirement for T&E study 4.1.11. 
The output medium is a report written in accordance with 
the format of the experimental design. The information to 
be included in the report is TBD. 
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6.1.12 Evaluation of Yield Models Relating 
Yield to Spectral and Spatial Appearance 
(Required by CAS; Category 2) 


This is the output requirement for T&E study 4.1.12. 
The output medium is a report written in accordance with 
the format of the experimental design. 


6.1.13 Evaluation of Cloud Detection and 
Data Dropout Algorithms 
(Required by CAMS/DAPTS; Category 2) 


This is an output requirement for T&E study 4.1.13. 
The output medium is a report written in accordance with 
the format of the experimental design. 


6.1.14 Evaluation of Alternative Analyst 
Interpretation Procedures 
(Required by CAMS/CAS; Category 2) 


This is an output requirement for T&E study 4.1.14. 
The output medium is a report written in accordance with 
the format of the experimental design. 


6.1.15 Evaluation of IMAGE100 System for LACIE Problem 
(Required by CAMS; Category 2) 


This is an output requirement for T&E study 4.1.15. 
The output medium shall be reports written in accordance 
with the format of the experimental designs. 


6.1.16 Evaluation of Acreage 
Determination Techniques 
(Required by CAS; Category 2) 


This is an output requirement for T&E study 4.1.16. 
The output medium shall be a report written in accordance 
with the format of the experimental design. 
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6.1.17 Evaluation of Production 
Determination Techniques 
(Required by CAS; Category 2) 


This is an output requirement for T&E study 4.1.17. 
The output medium shall be a report written In accordance 
with the format of the experimental design. 


6.1.18 Evaluation of Registration Techniques 
(Required by CAMS/DAPTS; Category 2) 


This is an output requirement for T&E study 4.1.18. 
The output medium shall be reports written in accordance 
with the format of the experimental design. 


6.1.19 Evaluation of SR&T Tasks 
(Required by CAMS; Category 3) 


This is the output requirement for T&E study 4.1.19. 
The output medium is report written in accordance with 
the formats of the experimental designs and in the manner 
described in 4.1.9 and 4.1.10. The detailed information 
to be included in these reports is TBD. 


6.1.20 Miscellaneous Output Products 
(Required by all Subsystems; Category 2) 


In addition to the specific task documents, TES shall 
provide 

a. Periodic, probably monthly, status report on all 
active tasks. 

b. Preliminary significant findings on an as-needed 
basis 

c. Publications from outside elements, where 
applicable 

d. Other TBD products 
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6.1.21 Investigation of Per Field Classifiers 
(Required by CAMS; Category 2) 


This is the output requirement for T&E study 4.1.20. 
The output medium is to be a report to be written in 
accordance with a TBD format. 


6.2 INTERIM SYSTEM 
(Required by CAMS; Category 2) 


This Is the output requirement for the interim T&E 
studies referred to in 4.2. In accordance with the quick- 
response, interim aspects of these tasks, the output medium 
will consist of internal memorandums in a TBD format. 
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7.0 INTERFACE REQUIREMENTS 




Each implementing organization shall comply with the 
interface requirements specified in the following documents 

a. Earth Resources Data Format Control Book , 

Document PHO 5^3, July 1973. 

b. GSFC/JSC Interface Control Document for LACIE , 
April 1974 . 


PRECEDING PAGE BLANK NOT FIT MED 
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8.0 OPERATIONAL REQUIREMENTS AFFECTING THE DESIGN 


8.1 THROUGHPUT REQUIREMENTS 
Not applicable. 


8.2 RESPONSE REQUIREMENTS 


Not applicable. 


8 . 3 RELIABILITY REQUIREMENTS 


TBD . 


8 . 4 SECURITY REQUIREMENTS 


A LACIE Security Plan shall be prepared by each 
organization designated with implementation responsibility. 
The plan for each implementing organization shall define 
the specific measures that will be utilized by the 
organization to comply with the LACIE security requirements. 
The LACIE security requirements shall be defined by the 
United States Department of Agriculture (USDA) and will 
be forwarded to the implementing organizations upon receipt 
by the LACIE Project Manager. Each implementing organiza- 
tion shall submit its plan for approval to the LACIE 
Level III Change Board within 90 days after receipt of 
the USDA requirements. 


8.5 DELIVERY REQUIREMENTS 


Not applicable. 
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8.6 QUALITY ASSURANCE REQUIREMENTS 


A LACIE Quality Assurance Plan shall be prepared 
for each organization designated with implementation 
responsibility. The plan shall cover a complete defini- 
tion of all quality assurance functions that will be 
implemented to assure maintenance of adequate quality 
levels during operation. The plans shall be submitted 
for approval to the LACIE Level III Change Board within 
90 days after the requirements documents are baselined. 


8.7 OTHER OPERATIONAL REQUIREMENTS 


Not applicable. 
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10.0 


RESEARCH REQUIREMENTS 


TBD. 
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11.0 TEST AND EVALUATION REQUIREMENTS 


TBD. 
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